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ABSTRACT, 


on'Tine)  estimation  of  parameters 


This  paper  discusses  the  recursive  (ori'TinF)  estimation  of  parcuneters' 
in  regression  and  autoregressive  integrated  moving  average  (ARIMA)  time  series 
models.  The  approach  which  is  adopted  uses  Kalman  filtering  techniques  to 
calculate  estimates  recursively.  This  approach  can  be  used  for  the  case  of 
constant  as  well  as  time  varying  parameters. 

In  the  first  section  the  linear  regression  model  is  considered  and  recur- 
sive estimates  of  the  parameters,  both  for  constant  and  time  varying  parameters, 
are  discussed.  Since  the  stochastic  model  for  the  parameters  over  time  will  be 
rarely  known,  simplifying  assumptions  have  to  be  made.  In  particular  a random 
walk  as  a model  for  time  varying  parameters  is  assumed  and  it  is  shown  how  one  can 
determine  whether  the  parameters  are  constant  or  changing  over  time. 

In  the  second  section  the  recursive  estimation  of  parameters  in  ARIMA 
models  is  considered.  If  moving  average  terms  are  present,  the  model  has  to  be 
linearized  and  the  Extended  Kalman  Filter  can  be  used  to  recursively  update 
the  parameter  estimates.  The  first  order  moving  average  model  is  discussed 
in  detail.  
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SIGNIFICANCE  AND  EXPLANATION 

The  problem  discussed  in  this  paper  is  the  following:  Observations 
up  to  time  n (of,  for  example,  business  data  or  missile  positions)  are 
available  and  one  wants  to  estimate  unknown  parameters  in  regression  or 

1 

autoregressive  integrated  moving  average  models  in  order  to  predict  futtire 
values . 

An  additional  observation  is  recorded.  The  question  becomes  how  to 
Ufxlate  the  parameter  estimates  from  the  previous  estimates  and  the  most  recent 
observation  without  storing  the  complete  past  history  of  the  data.  The 
answer  to  this  question  will  depend  on  whether  the  parameters  in  these  models 
are  assumed  constant  or  whether  they  themselves  follow  a given  stochastic 
process . 

This  recursive  estimation  procedure  (which  is  sometimes  called  on-line 
estimation  or  parameter  tracking)  is  important  if  the  observations  become 
available  sequentially  in  time.  It  has  applications  in  economics  and 
business,  where  economic  indicators  and  sales  data  are  updated  every  week, 
month  or  quarter;  it  can  also  be  applied  to  the  control  of  satellites  or 
ballistic  missiles  where  the  position  in  space  is  recorded  every  few  seconds. 


The  responsibility  for  the  wording  and  views  expressed  in  this  descriptive 
summary  lies  with  MRC,  and  not  with  the  author  of  this  report. 
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A RECURSIVE  APPROACH  TO  PARAMETER  ESTIMATION  IN  REGRESSION 


AND  TIME  SERIES  MODELS^ 
Johannes  Ledolter 


1.  Recurs 


51.1:  We 


1 ve  parameter  estimation  in  reqr ess  ion  models 
consider  n observations  from  the  reqression  model 


(1.1) 


where:  is  the  dependent  variable  at  time  t 

h = (li  ....,h  ) is  a set  of  m independent  variables  at  time  t 

t It  mt 

2 

{a^t  is  a sequence  of  independent  normal  random  variables  N(0,o  ). 
Standard  reqression  procedures  assume  that  the  parameters  are  constant  over  time 
(Bj.  = B for  all  t)  . In  this  case  the  least  squares  (maximum  lilcelihood)  estimates 
of  6,  qiven  all  the  data  up  to  arrf  includinq  time  n,  are 

B = ’z'"’ 


where 


(n) 

z 

■•'V 

h‘"> 

••'•Jn 

A recursive  version  of  least  squares  estimates  in  regression  models  was  first 

qiven  by  Placlcett  18).  He  proves  that  the  parameter  estimates  at  time  n are  linear 

combinations  of  the  estimates  at  time  n - 1 and  the  prediction  error  z - h*B 

n -n-n-l 

Plac)?ett's  solution  can  be  shown  to  be  a special  case  of  a more  qeneral  procedure 
which,  in  the  engineering  literature,  is  Itnown  under  the  name  of  Kalman  filtering 
(Kalman  !4),  Kalman  and  Bucy  (51).  In  the  following  section  (51.2)  we  given  a brief 
review  of  this  approach.  A similar  discussion  is  given  by  Duncan  and  Horn  (21. 
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§^L_.2:  We  consid(?r  the  r«>qression  mcxlel  in  (1.1) 


(1.2) 


6 = TB  + € , 

•t*l  t t+1 


with  param*?ters  which  ctiamic.  accordirif)  to  a multiplt*  (m-d imonsional)  autoreqrossivc  mcjdel 

(1.3) 

(e  ) is  a sequence  of  independent  normal  rantlom  variables  H (O.o^Si)  . Furthermore 
c nt 

it  is  assumed  that  and  a^  are  i ndeq^rndent ; ~ 

Assuminq  that  the  initial  parameter  vector  at  time  0,  follows  a normal 


distribution  with  exfjectat ion  ?'q|q  ^™1  covariance  matrix 

shown  that: 


it  is  easily 


B 1 ' 

r 

•t|t-l 

2 

.»?;5tit-i. 

t t-1 


P , h 
t t-l't 


h'P  I , 1+h’P  I ,h 
t t t-1  -t  t t-1  t 


and 


It  follows  from  properties  of  the  normal  d istr iljut ion  that 

■1  ■ 6 , ,) 
t t t-1 


B 1 

= B 1 

+ 

K , (z 

'tit 

-t| 

t-1 

t-1  t 

1 

= f"  1 

_ 

K ,h'P 

tit 

1 t-1 

t-1  t 1 

K , 

= f 1 

(1  t h'P 

t-1 

t| 

! t-1  t 

■t 

.h  ) 


-1 


is  called  Kalman  gain.  Furthermore  it  follows  from  (1.4)  that 

P(B  , ,o^,n,T,B,  I ,P„|  ) ^ N (B  ,1  ,o^p  I ) 

*^1+1  - -OlO  0|0  m t+llt  t+llt 


(1  .4) 

(1.5) 

(1.6d) 

(1.6b) 

(1.6c) 


where 


B 


ttllt 


T6  I 
-t  t 


"tnlt  ■ "’t|  J'  ^ " • 


(1 .6d) 
(1.6e) 


Equations  ( 1 . 6a) - ( 1 .6e)  can  be  used  to  calculate  the  parameter  estimates  recursively 
as  each  new  observation  becomes  available  in  time.  B^|^  and  starting  values 

for  the  recursive  estimation  procedure  in  equations  (1.6).  To  reflect  ignorance  about  the 


-2- 


I 


parameters  at  time  zero  (before  observations  become  available),  the  matrix  P i 

o|o 

is  usually  chosen  diaqonal  with  large  values  in  its  diagonal.  If  n is  moderately 


t 


larqe,  the  initial  choice  of  ^'qIq  will  be  dominated  by  the  information  from  the  data. 


Si  ♦ 3 • f’ot  the  case  T » I and  U = 0 the  alxDve  procedure  specializes  to  the  recursive 
least  squares  algorithm  discussed  by  Plackett  (ft), 

B 


:t|t  = ^-i|t-l  ^ " '?tVl|t-l'‘t>'‘Vilt-l''t‘=t  - "l^^-llt-l' 


(l-7a) 


and 


^t|t  't-ijt-i  * ’^t't-ijt-i^’  ''t-i I t-i'^t^^'t-i it-i  ■ 


(1.7b) 


.^.^-'.1*  Kquation  (1.3)  specifies  an  autoreqrc*saive  procffss  as  model  for  the  time  varyinq 
parameters.  This,  however,  is  not  a serious  restriction  since  by  the  introduction 
of  additional  state  variables  any  autoregressive  integrated  moving  average  (ARIMA) 
process  can  be  brought  into  this  form. 

by  substitution  it  can  be  shown  that  the  regression  model  (1.2)  in  which  the 
parametf*rs  follow  a multiple  AHIMA  process 


(r  - - 


- R'  ) B 

P '-tM 


(I  - h^b 


- H H^)e 


t+1 


(1  .8) 


where  I is  the  identity  matrix  and  I*  .....t  , . . . ,H  are  known  |m  « ml  matrices, 

1 p 1 g 


can  be  written  as: 


Vl ' 

'*’1 

1 

'2ttl 

Nk-1  )m 

* 

* 

- 

• 

'2t 

+ 

'2tfl 

• 

_-kt+l_ 

0 • • • 0 

* 

* 

_*kt+l_ 

(1.9) 


and 


?■) 


?2t 


?ktJ 


(I.IO) 


-3- 


k = max{p,q  ^ l};  t , = 0 for  j > p;  = 0 for  j ^ q:  f,,  ,,  is  the  (k  - i)m 

3 3 j -1  (k-l)m 

* 

identity  matrix;  f,  , = -W.  , for  2 ^ j < k. 

'3t+l  j-l't+1  - - 

The  recursive  procedures  given  in  (1.6)  can  be  applied  to  the  system  described 
by  equations  (1.9)  and  (I.IO). 

§1.5:  The  model  in  ecjuation  (1.4)  assumes  that  the  parameters  vary  around  the 

origin.  In  general,  however,  parameters  will  vary  around  a level  B which  will  be 
different  from  zero. 

Redefining  the  matrix  T will,  however,  allow  for  ^ ^ 0: 

IT  T - ll 


-tu 

.^t+1 

0 


z^  = fh/O'l 
t -t- 


I 


n 

'b' 

’l' 

-t 

+ 

B. 

0 

J 

~t 

. _ 

-t+1 


(1.11) 


(1.12) 


§jl^.6:  A rarxiom  walk  model  for  the  parameters 

The  recursive  estimation  procedures  in  equations  (1.6)  are  very  general. 
Theoretically,  they  can  be  used  to  u[xlate  regression  estimates  for  any  given  ARIMA 
nxxlel  for  the  parameters. 

In  [practice,  however,  the  model  for  the  parameters  is  rarely  known  and  simplify- 
ing assumptions  abf:ut  this  model  have  to  be  made.  A particularly  useful  and  simple 
m<xiel  for  time  varying  parameters  is  the  random  walk  model. 


B = B + « 

-t+1  -t  -t+l 


(1.13) 


which  equivalently  can  be  written  as 


B 


= 6 + y e 

t+1  -0  -t  + 1-3 

3=0 


It  represents  the  parameter  at  time  t+1  as  a sum  of  the  parameter  at  time  0 and 

a cumulative  sum  of  independent  random  variables.  The  random  walk  model  allows  for 

sBKjoth  changes  In  the  parameters.  The  variability  of  the  parameters  depends  on  the 

2 

covariance  matrix  “ o 11.  If  11  = 0 the  parameters  are  constant  over  time. 
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Usinq  the  observations  up  to  and  including  time  n,  we  address  the  question 
whether  the  parameters  in  the  regression  model  (1.2)  can  be  considered  constant,  or 
whether  there  is  indication  of  time  changing  parameters  iU  ^0  in  the  random  walk 


Setting  T = r,  it  follows  from  equation  (1.4): 


f = 1 + h'P  I h . 
t -t  t t-l't 


Furthermore, 


a"rTf  exp.  --4  I 

t=l  2o  t=i 


and  Pq|q  ean  be  treated  as  piarameters.  However,  if  there  is  little  prior 
information  about  6^,  the  matrix  Pg|Q  in  i^’e  recursions  (1.6)  can  be  chosen 
diagonal  with  large  entries  in  its  diagonal;  for  the  initial  Bg|g  one  can  choose  the 
zero  vector. 

2 

The  log  lilcelihood  function  of  the  parameters  a and  ft  is  given  by 


f (a^,n(z^"\H*"^)“  - n log  ~ J I log  f^  - — ^ 


? - ^t^tlt-l’' 


The  ML  estimate  of  o is  given  by 


-2  1 ? - ^;gtit-i’ 

” ‘"t-i  ^ 


and  the  concentrated  likelihood 


l^(n|z*"*  ) « - n log  ~ ^ I log  f^  . 
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Nonlinear  optimization  techniques  can  be  used  to  derive  the  Mb  estimate  of  the 

covariance  matrix  II.  Approximate  (1  - '■j)100  percent  (.'Onfidence  limits  for  the 
k(k  ♦ 1) 


elements  in  U are  given  by  the  contours  of 


2 

2 . 2 
whore  Xj-i'-Ji)  is  the  upper  u {>ercf*nt  cut  off  value  of  a x distribution  with  f 

deqrees  of  freedom. 

§1.7;  An  example 

We  consider  the  model 


(n)  ..<n). 


1 2 


2 = 6^  + a 

t t t 


V(a^.)  = a 


V(6^)  = ko 


(i)  The  up-dating  relations  (1.6)  simpjlify  to 

®tit  = ^t-Ht-i  ^ <Vi|t-i  ^ ^ ^>"<Viit-i 

"tit  = 'Viit-1  * ‘^'/‘Viit-i  ^ ^ 

and  the  concentrated  log  likelihood  function  in  (1.18) 


i (k|z^"*)»  - n log  c - ~ log  f 

t=l 

where 


^ = "t-l(t-l  " ' 


and 


k > 0 


" - ^-i|t-i> 


(1.19a) 

(1.19b) 

(1.20a) 

(1.20b) 

(1.21) 


-2  1 ? - \-llt-l> 

n : 


■■  t=l  "t-llt-l  ^ ^ 

The  log  likelihood  function  can  be  calculated  and  plotted  for  various  k.  The  maximum 
likelihood  estimate  k can  be  found  and  apj-roximate  (1  - (j)100%  confidence  limits 
are  given  by  the  solution  to 


|t  (t!z‘"')  - » (klz'"’)| 
' c c • 
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i . 


(1.22) 


1 


2.  Recursive  estimation  of  parameters  in  APTMA  time  series  models 

52^1:  Autoreqressi VC  integrated  mtivinq  averages  (ARIMA)  time  series  models  are  described 

by  tiie  stochastic  difference  equation 


(1  - I'jB  - 


- 4.  B'  ) (z^  - U)  = (1  - ((  B - 
!'  t 1 


- e B'^)a 


(2.1) 


where;  Ii  is  f.iui  backshift  om^rator;  B ■/.  « z 

t t-m 

rj 

is  a stationary  difference  of  the  original  observations,  = (1  - B) 
the  roots  of  -1*  (Ii)  = i - = 0 and 


« (B)  = 1 - 0,B  - 


- 0 = 0 are  assumed  outside  the  unit  circle; 


furthermore  4- (B)  - 0 and  0(B)  = 0 have  no  common  roots. 

2 

fa^}  is  a sequence  of  indei^endent  N(0,a  ) random  variables. 


§2.2:  Given  observations  z 


(n) 


(z  ,z^,...,z  )',  maximum  likelihood  estimates  for 
12  n 


the  parameters  fi*  = ( u , 4*  » • • • / 6^,..., 6^)  can  be  derived.  Box  and  Jenkins  [11, 

bjung  and  Box  (61  discuss  the  derivation  of  the  exact  likelihood  function  of  the 
pdram#?ters  for  this  class  of  models.  In  qeneral,  the  exact  likelihood  function  is  non- 
linear in  the  parameters  and  iterative  maximization  techniques  have  to  be  used  for 
the  derivation  of  the  ML  estimates.  Ljung  and  Box  (6]  prO{x5se  a general  method  for 
the  calculation  of  the  likelihood  function. 

For  pure  autoregressive  processes  the  derivation  can  be  simplified. 


The  ML  estimates  of  ~ (4>j^»4'2» 


) ' can  be  approximated  by  the  least  squares 


estimates. 


^2._3:  Ml.  estimates  for  the  parameters  in  AkIMA  models  are  nonrecursive  in  nature. 

If  data  is  collected  sequentially  (such  as  quarterly  or  monthly  economic  data,  sales 
data,  hourly  ix>llutant  measurtiments)  one  would,  however,  prefer  recursive  estimation 
procedures  which  ujidate  the  values  of  the  parameters  as  each  new  observation  becomes 
aval lable. 

Uixiating  (tracking)  the  parameters  should  not  be  confused  with  updating  the  fore- 
casts as  discussed  by  Box  and  Jenkins  [1,  Chapter  5).  U|x3ating  of  the  forecasts 
assumes  constant  parameters  and  calculates  the  revised  forecast  as  each  new  observation 
is  made  available.  Tracking  the  parameters  ufxiates  the  parameter  estimates. 


A 


i 
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§2.4:  Linearization  of  the  model 

The  AKIMA  model  in  (2.1)  is  nonlinear  in  the  parameters.  To  derive  recursive 
estimates  the  model  has  to  be  linearized.  This  can  be  achieved  by  expanding  the  APIMA 
model 


j = g t-  y 11,(41, 0)(z  . - u)  + a^ 

t • . 1 ' ' t-]  t 

J>1 


where  the  weights  in  (2.2)  are  given  by  the  coefficients  in 


ir(B)  = 1 - y 

in  a Taylor  series  around  some  reference  value 
bef ine 


_ l|i(B) 

■ 6(B) 

6 = ' . 


(2.2) 


(2.3) 


f(g;z^*'  = u + y ■n.(i(i,e)(z  . - g) 

j>l  3 - - t-D 


then. 


^ ^ ^ (u  - U) 


U.  (6)  + y (4|.  - it.  )v.  (6)  + ) (0.  - 
1=1  3 = 1 


(2.4) 


0j)w^t<§>  (2.5) 


where 


''it'?’ 


(1  i < p) 


w (6)  = f (B:  z ) I 

jt  d0 . ' 'b=6 

3 


(1  £ j 1 q)  . 


(2.6) 


z^  - f(B;z'^  ^’)  = is  the  one  step  ahead  forecast  error  for  the  ARIMA  model 

with  parameters  6. 

It  can  be  easily  shown  that 


u^(B)  « (1 

)/(i 

t ' 

i p 

(1  - is  - 

...  - 4 bP)v.,.(6)  = 

1 

p it  - 

(1  - 6 B - 

- 0 b'’)w.^(6)  = 

1 

q 3t  - 

(2.7) 
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Usin<?  this  expansion  th«'  n**neral  AHTMA  ino(h‘l  (2.1)  with  t imi-  varyirKi  f i c i **nt  s 


t-  f 1 


TB  ► c ; 
t t M 


\:c  f * s n u 
t t 


(2.H) 


can  hv  rjxpandud  around  the  trajoctory  ^ ^ = tB^  and  represented  as 


' h-  (P,^  - P,^)  . .1^^ 


“'■tu  - - 't.>  * Sh 


eli'/iH’iits  in  *')  = 

t.'Xprtissions  In  (2.7)  witii  P rnplacccl  by  f' 


J . /W  ,...,w  )'  arn  tjiuen  |jy  thn 

(4  It  'it 


(2.9) 


•Summ.ir  i z i nrj , it  is  shown  that  thn  nonlinear  AlviMA  model  with  time  varyin't  I'ara- 
meter.s  (the  special  case  of  constant  parameters  is  qiven  wtien  T = I and  Si  = 0)  can 
be  linearized  and  ai>proximated  as  in  (2,9).  (2.';i)  is  linear  in  tht?  mea.surement  devia- 
tions z^  - fiP^iz^*"  ^*)  and  in  the  trajeertory  (state)  deviations  - B^. 

Given  a trajectory  and  observations  z^^^,  one  can  calculate  tlie  measurement 

deviations  z^  - f(B^;z^  and  use  the  Kalman  filter  equations  in  (1.6)  to  derive 


an  estimate  deviations  from  the  trajectory  (dB)^  ~ ^t  ~ ^t" 

estimate  for  p.  is  then  given  by  p , = B + (dP)  , . 

-t  't|t  t t|t 

An  obvious  choice  for  the  reference  trajectory  is  = B^,  where  B^  is  the 

prior  estimate  of  the  parameter  vector  B.  However,  if  our  prior  estimate  is  poor, 
the  state  deviations  B^.  ~ B^.  can  become  large  and  the  linearity  assumption  used  in 
tlirr  exfiansion  (2.5)  may  be  violated. 

However,  the  following  improvement  can  be  made;  the  AHIMA  model  (2.9)  can  be 
linearized  about  each  new  estimate  as  estimates  become  available  in  time.  At  time  t 
(after  has  become  available)  we  linearize  around  ^t|t'  Processing  can 

u/xlate  the  parameter  vector,  derive  ®ru|t4i  this  estimate  for  the  next 

linrrarization.  This  procedure  uses  a better  estimate  for  the  trajectory  as  soon  as 
one  becomes  available,  and  large  errors  alxiut  the  trajectory  due  to  bad  a priori 
estimates  are  not  allowed  to  propagate. 
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it  follows  f rrim  tho  rc?cursiv<'  1 i nocir i zat i(<n  of  tho  AhflM^  modol  that 


UU<) 


tf i t 


(2.10) 


and 


It*!  ^'tU  It  + l 


t * 1 t. 


(2.11) 


u.sitiq  the  Kdlmjn  filt(?r  c-(juat  ions  for  t)ic  linearized  system  (2.'J)  , the  U|jdatinq  etjua- 
tions  arrt  fhr»rororo  qivcfn  by: 


^tult  = '’■'^tlt  ■ 

''tMlt  " '‘'''tit'’"  * •'  ' 

\it  = etit-i  ^ Vi<\  - ' 


I'  I = I'  I - K h'  (ft  I ; ' ) I' 

t|t  t t-1  t-1  't  t-l 


t|t-l  ' 


K = I'  I h‘  (B  I 
t-1  t t-1  t t-1 


i'*""”)  II  * ).• 


t,t-l>-t,t-.-"'"»-' 


(2.12a) 
(2- J2b) 
(2.12c) 

(2.1  2d) 

(2.120) 


/ r»  the  onqinoorinq  literature*  tho  qonoral  procodure  of  rccursivoly  linoarizinq  non- 
lir»e.*ar  mexiols  is  known  under  tho  name  Kxtr;ndod_  Kalman  F'iltorin^.  Se*o  Jazwinski  I'll 
for  further  roforenco. 

§2.5:  KxampU;:  Rocuraivo  paramotor  ostimation  for  tho  first  order  rw^vinq  avoraqo 

process  with  constant  parameter 

(i)  Tho  first  order  movinq  avoraqo  process  (for  u = 0)  is  qivon  by 

^ ^ - “"t-1  • 

It  is,  fr>r  example,  shown  by  hjunq  and  Box  (b]  that  the  ML  estimate  of  0 for  the 
first  order  movinq  averaqe  model  is  obtained  by  minimizinq 

1_ 

1,(0)  « •(  ) O"-’  i S(0) 


(2.13) 


{n  ti 

j 

j=0  ^ 


(2.14) 


whore 


S(0)  - J af(n)  , 

t^o 


SqIO) 


-1  n 

I 0^ 

1 1 "T'  0"'"'^’ 

U-0  J 
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In  the  second  section  the  recursive  estimation  of  {parameters  in  ARIMA 
models  is  considered.  If  movinq  averaqe  terms  are  present,  the  model  has  to 
be  linearized  and  the  Extended  Kalman  Filter  can  be  used  to  rficursively 
u{pdate  the  parameter  estimates.  The  first  order  movinq  averaqe  mfpdel  is 
discussed  in  detail. 


I 


